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Abstract 

The  main  goal  of  our  research  program  is  to  develop  the  mathematical  concepts  and  algorithms  for 
performing  a  wide  range  of  operations  on  motions  of  physical  objects,  including:  capture,  representation, 
synthesis,  segmentation,  editing,  optimization,  simulation,  and  execution.  In  addition,  we  have  focused  on 
the  support  tools  that  are  critical  for  understanding  and  analyzing  physical  and  synthetic  agents  interacting 
in  a  dynamic  environment.  These  include  the  underlying  data  structures,  the  architecture,  and  the  analysis 
and  synthesis  tools.  The  term  Algorithmics  of  Motion  refers  to  the  overarching  framework,  the 
mathematical  concepts,  the  algorithms  and  the  tools  for  this  work.  This  document  presents  an  executive 
summary  of  our  work  over  the  last  five  years  from  October  1995  to  September  2000. 

1  Introduction 


Algorithmics  of  Motion  is  a  multi-university  research  initiative  between  two  universities:  The  University  of 
Pennsylvania  and  Stanford  University.  The  main  goal  of  the  MURI  is  to  bring  together  a  group  of 
distinguished  researchers  with  complementary  expertise  to  develop  the  framework,  mathematical  concepts, 
algorithms,  data  structures,  architecture,  and  analysis  tools  for  performing  a  wide  range  of  operations  on 
motions  of  physical  objects  including:  capture,  representation,  synthesis,  segmentation,  editing, 
optimization,  simulation,  and  execution. 


1  Principal  Investigator  before  December  1,  1998,  presently  at  the  National  Science  Foundation. 

2  Principal  Investigator  after  December  1,  1998,  kumargcis .  upenn .  edu. 


The  main  investigators  (all  faculty  members)  in  alphabetical  order  are: 


■  Ruzena  Bajcsy  (University  of  Pennsylvania) 

■  Kostas  Daniilidis  (University  of  Pennsylvania) 

■  Leonidas  Guibas  (Stanford  University) 

■  Vijay  Kumar  (University  of  Pennsylvania) 

■  Jean-Claude  Latombe(Stanford  University) 

■  Max  Mintz  (University  of  Pennsylvania) 

■  Rajeev  Motwani  (Stanford  University) 

■  Jim  Ostrowski  (University  of  Pennsylvania) 

■  Camillo  Taylor  (University  of  Pennsylvania) 

■  Carlo  Tomasi  (Stanford  University) 


2  Our  Goals  and  Vision 

The  main  goals  of  our  research  program  are  the  framework,  mathematical  concepts,  and  algorithms,  for 
performing  a  wide  range  of  operations  on  motions  of  physical  objects,  including:  capture,  representation, 
synthesis,  segmentation,  optimization,  simulation,  and  execution.  An  important  aspect  of  this  research  is 
the  suite  of  support  tools  that  are  critical  for  understanding  and  analyzing  physical  and  synthetic  agents 
interacting  in  a  dynamic  environment. 

The  world  is  governed  by  fundamental  laws  of  physics  that  describe  the  relationships  between  the  position 
and  motion  of  objects  and  the  forces  that  act  on  them.  In  order  to  understand  such  physical  agents,  it  is 
necessary  to  have  the  ability  to  model,  analyze  and  control  such  machines  as  cruise  missiles,  computer- 
controlled  tanks,  and  autonomous  and  semi-autonomous  agents  at  the  micro,  meso,  and  macro  scale.  An 
important  aspect  of  the  functioning  of  these  machines  is  their  interaction  with  a  dynamically  changing 
environment.  As  computers  and  computer  networks  become  ubiquitous,  we  now  have  the  ability  to  create 
synthetic  human  agents  that  can  interact  with  virtual  models  of  physical  objects  (including  machines)  in  a 
virtual  world.  However,  even  here,  it  is  important  that  the  motion  and  interaction  of  the  synthetic  agents 
conform  to  the  laws  of  physics  and  the  dynamic  constraints  that  are  representative  of  the  real  world.  Finally 
as  computers  and  computer  controlled  machines  are  designed  to  augment  a  human  user’s  capabilities,  the 
relative  motions  and  forces  involved  in  the  interaction  between  the  human  agent  with  other  physical  agents 
becomes  critical.  Our  research  is  driven  by  the  fundamental  problems  that  arise  in  capturing  and 
representing  motion,  synthesizing  and  planning  motion,  and  executing  or  controlling  a  planned  motion. 

While  the  objective  of  this  research  initiative  is  to  solve  fundamental  research  problems,  our  thrust  areas 
and  the  problems  are  driven  by  many  important  applications. 

Military  Applications: 

■  Autonomous  or  semi-autonomous  vehicles  traversing  a  hard,  uneven  terrain,  and  transporting 
heavy  equipment; 

■  Support  for  marines  landing  on  hostile  beaches  and  mine  laden  ports;  and 

■  Support  for  special  operation  forces  in  urban  warfare,  monitoring  illegal  activities,  scouting 
buildings,  clandestine  access  to  buildings,  and  reconnaissance. 

Civilian  Applications: 

■  Manufacturing  applications:  designing  collision-free  paths  for  assembly,  reverse  engineering  of 
complex  three-dimensional  products,  and  design  of  one-of-a-kind  products  (assistive  devices  for 
people  with  disabilities  and  aging  people); 

■  Medical  applications:  synthesis  of  new  drugs,  computer  controlled  surgery,  and  tele-immersive 
technologies  for  tele-surgery. 


3  Technical  Objectives 

Our  technical  objectives  and  research  agenda  are  addressed  under  the  important  thrust  areas  that  we  have 
identified.  These  thrust  areas  can  be  divided  into  the  three  different  foci  of  the  initiative. 


3.1  Motion  Capture 


Motion  capture  entails  sensing  and  measuring  attributes  of  the  external  world,  analyzing  the  measurements, 
and  representing  this  information  efficiently  in  a  form  that  can  be  used  for  motion  synthesis  and  execution. 

The  key  thrust  areas  under  motion  capture  are: 

■  Recovery  of  3-dimensional  environments; 

■  Image  motion  analysis;  and 

■  Vision-based  reconstruction  for  mobile  robots  and  telerobotics. 


3.2  Motion  Synthesis  and  Planning 


Motion  synthesis  refers  to  the  techniques  used  to  automatically  generate  motion  plans  (e.g.,  collision-free 
paths  based  on  geometric  information,  or  sensor-based  motion  strategies)  from  spatial,  geometric,  and 
temporal  data.  Plans  may  be  generated  from  scratch  or  by  modifying  or  editing  pre-existing  plans  to 
optimize  over  a  set  of  meaningful  criteria. 

The  key  thrust  areas  are: 

■  Advanced  motion  planning; 

■  Sensor-based  motion  planning;  and 

■  Sensor  fusion  for  tracking  and  interception  of  moving  targets. 


3.3  Motion  Execution 

Motion  execution  is  the  controlled  generation  of  the  synthesized  motion  in  a  graphically  simulated  or  in  a 
physical  world.  Motion  execution  in  the  real  world  requires  a  tight  coupling  with  motion  capture,  modeling 
and  representation  of  motion,  synthesis,  and  planning.  Motion  execution  in  the  graphical  world  entails 
representation  of  motions,  synthesis,  planning,  and  rendering. 

The  key  thrust  areas  are: 

■  Collision  detection  using  kinetic  data  structures; 

■  Control  and  coordination  of  multiple  robots;  and 

■  Control  of  advanced  locomotion  systems. 

The  key  accomplishments  and  the  plans  for  future  research  in  each  thrust  area  are  briefly  described  in  the 
following  sections.  The  effort  in  the  thrust  areas  is  summarized  in  Table  1. 


4  Technical  Approach 


Our  technical  approach  is  anchored  in  a  problem  solving  loop.  As  stated  in  our  technical  objectives,  we  are 

interested  in  a  class  of  problems  that  involve  agents  interacting  with  dynamically  changing  environments. 

Hence,  the  problem  solving  loop  consists  of: 

(1)  Using  mathematical  tools  to  create  conceptual  models  that  we  believe  best  describes  the  problem  or 
the  sub-problem.  These  models  can  be  equations  and/or  algorithms. 

(2)  Use  these  models  to  develop  computer  simulations  to  test  the  validity  of  our  problem  solving. 

(3)  If  the  problem  addresses  an  interaction  of  physical  agent  and/or  physical  environment,  we  build  a 
physical  model  to  verify  our  initial  assumptions  and  conceptual  models. 

(4)  Therefore  we  conduct  experiments  for  validation  and  benchmarking. 

(5)  If  these  experiments  agree  with  our  expectations/problem  specifications  then  we  are  done,  otherwise 
we  refine  the  conceptual  model  and  repeat  the  steps  2-4. 

This  methodology  is  a  common  theme  in  all  the  research  presentations. 


Table  1  The  Key  Thrust  Areas,  Principal  Researchers,  and  Leveraged  Funding  and  Collaborations, 


Thrust  Areas 

Principal 

Researchers 

Collaborations  &  Funding  Sources 

MOTION  CAPTURE 

Recovery  of  3-D  Environment 

Daniilidis,  Bajcsy 

DARPA 

Image  Motion  Analysis 

Tomasi 

GRC,  Intel,  Interval,  NSF,  Sony 

Reconstruction  for  Mobile  Robots  and 
Teleoperation 

Taylor 

GM  and  DARPA 

Medical  Imaging,  Neuro-anatomy  and 

Pattern  Matching 

Bajcsy,  Gee 

NIH 

3-D  Model  Construction 

Bajcsy,  Latombe 

NSF 

Interfaces  for  People  with  Disabilities 

NSF 

MOTION  SYNTHESIS 

Randomized  Motion  Planning 

Latombe,  Motwani 

Intel,  GE,  and  GM 

Motion  Planning  with  Visibility  Constraints 

Guibas,  Latombe 

NSF,  AASERT,  Intel,  DARPA 

Sensor  Fusion  for  Control  and  Planning 

Mintz,  Kumar, 

Taylor,  Ostrowski 

DARPA 

Sensor-Based  Motion  Planning 

Kumar,  Mintz, 
Ostrowski 

GM,  NSF,  DoE,  Harvard  Univ. 

MOTION  EXECUTION 

Kinetic  Data  Structures 

Guibas 

AASERT,  NSF,  Duke  Univ.,  Intel 

Control  of  Multiple  Robots 

Kumar,  Ostrowski 

NSF,  DoE,  Harvard,  Tokheim,  Honda 

Novel  Sensors  and  Vision  Based  Control 

Bajcsy,  Taylor 

GM 

Motion  Control  over  the  Internet 

Bajcsy,  Latombe 

NSF 

Adaptive  Locomotion  Systems 

Ostrowski,  Kumar 

NSF  and  DoE 

Software  for  Multi-robot  Coordination 

Mintz,  Kumar, 
Ostrowski,  Pappas, 
Taylor 

DARPA 

Design,  Implementation,  and  Validation  of 
Embedded  Software 

Kumar,  Pappas 

DARPA 

5  Synergistic  Efforts 


5.1  Collaboration  Between  Researchers  at  the  University  of 
Pennsylvania  and  Stanford  University 

Our  interaction  has  consisted  of  formal  visits  by  the  P.I.s  on  a  regular  basis.  For  example.  Professors 

Latombe,  Guibas  and  Tomasi  have  visited  Penn,  while  Professors  Bajcsy,  Daniilidis,  Kumar,  Metaxas,  and 

Taylor  have  visited  Stanford.  In  addition,  the  principal  post-doctoral  students  have  exchanged  visits. 

Messrs,  Lavalle  and  Gonzales  from  Stanford  have  visited  Penn,  while  Desai  and  Goktas  have  visited 

Stanford  to  discuss  MURI  collaborations. 

In  addition  to  the  formal  planned  visits,  we  have  a  number  of  meetings  at  such  premier  conferences  as 

ICRA,  CVPR,  ISRR,  and  IROS.  As  the  conference  proceedings  publications  list  shows,  our  MURI  is  well 

represented  at  such  conferences. 

We  list  below  examples  of  collaboration  between  MURI  researchers. 

*  The  research  conducted  by  Bajcsy  and  Latombe  on  the  construction  of  3-D  models  of  robot’s 
environments  has  resulted  in  an  NSF  proposal  to  develop  jointly  a  software  system  that  will  enable  a 
robot  to  navigate,  make  sensing  operations,  and  build  a  3-D  model  of  its  environment.  The  focus  of 
this  proposal  is  on  dealing  with  uncertainty  (errors  in  robot  localization,  uncertainty  in  sensing).  NSF 
has  awarded  us  a  one-year  grant  (approximately  $120,000). 

■  Kumar  and  Latombe  have  collaborated  on  motion  planning  algorithms.  Latombe’s  methods  are  used 
to  generate  an  initial  plan  that  satisfies  geometric  constraints  while  Kumar’s  methods  are  used  to  refine 
the  plan  to  satisfy  kinematic  and  dynamic  constraints  while  optimizing  a  cost  function. 

■  Lavalle,  Kumar,  and  Mintz  have  had  a  fruitful  collaboration  on  game-theoretic  methods  for  motion 
planning  in  the  presence  of  uncertainty. 

■  Daniilidis,  Tomasi,  and  Taylor  have  had  discussions  on  image  analysis  and  geometric  problems  in 
computer  vision. 

■  Guibas  and  Kumar  have  started  discussions  on  the  complementary  properties  of  kinetic  data  structures 
and  control  graphs,  and  their  applications  to  controlling  multiple  autonomous  agents. 

*  Daniilidis,  Tomasi,  and  Taylor  were  together  with  Sastry,  Kosecka,  and  Ma  the  instructors  of  the 
ICRA  Tutorial  on  Structure  from  Motion,  San  Franscisco,  April  28,  2000. 


5.2  Collaboration  with  Other  Multi-University  Research  Initiatives 

In  addition  to  collaboration  within  the  MURI,  we  have  active  collaborations  with  other  MURIs  funded  by 

DoD.  We  provide  some  examples  of  such  collaborations  below. 

■  Leonidas  Guibas  and  Pankaj  Agarwal  (Brown/Duke/Johns  Hopkins  MURI)  have  been  collaborating 
closely  for  several  years  now.  In  particular  they  have  worked  extensively  on  static  and  kinetic  binary 
space  partitions  and  their  applications  in  graphics.  A  student  of  Agarwal,  T.  Murali,  is  now  at  Stanford 
as  a  post-doc  under  Guibas  and  Latombe.  Guibas  and  Agarwal  have  also  collaborated  on  several 
problems  involving  moving  points  or  graphs  with  changing  edge  weights. 


■  Bajcsy,  Daniilidis,  Kumar,  and  Ostrowski  have  an  ongoing  interaction  with  S.  Sastry,  J.  Kosecka,  R. 
Alur  and  others  (Berkley/Stanford/Cornell  MURI)  on  studies  of  hybrid  systems  and  vision  based 
control. 

■  Kumar  has  started  discussion  with  Robert  Howe  (Stanford/Harvard/Johns  Hopkins  MURI  funded  by 
ONR)  biomimetic  robots.  A  student  of  his,  Jaydev  Desai,  worked  as  a  post-doctoral  researcher  at 
Harvard.  Kumar  spent  his  sabbatical  (Fall  1999)  at  Johns  Hopkins  interacting  with  the  MURI  group 
there. 


6  Significant  Accomplishments 

6.1  Motion  Capture 

6.1.1  Recovery  of  3D  environments 

We  (Daniilidis  et  at)  have  developed  an  algorithm  for  the  recovery  of  3D-scenes  with  a  moving  camera 
using  minimal  calibration.  We  developed  an  algorithm  for  estimating  camera  parameters  and  registering 
several  stereo  views  based  only  on  a  few  landmarks.  As  a  member  of  the  National  Tele-immersion 
Initiative  we  set  as  our  goal  to  make  feasible  an  immersive  sense  of  remote  presence,  so  compelling  that  it 
instantaneously  creates  the  illusion  of  being  near  to  people  and  objects  which  is  physically  miles  away.  The 
key  issues  here  are  the  real-time  scene  acquisition  and  the  efficient  representation  of  scene  descriptions.  We 
have  developed  algorithms  for  view-independent  scene  reconstruction  and  motion-predicted  stereo 
matching  (ISAR  2000,  ECCV  2000,  ICPR  2000).  During  an  unprecedented  session  on  May  9,  2000,  we 
demonstrated  a  three-party  tele-immersion  system  with  real-time  on-line  3D  reconstructions  of  the 
participating  persons  transmitted  from  Philadelphia  and  Armonk  over  Intemet2,  and  projected  in  a  spatially 
augmented  display  designed  in  Chapel  Hill. 


Figure  1  The  National  Tele-immersion  Initiative.  A  viewer  in  Chapel  Hill,  North  Carolina,  is  able  to 
interact  with  collaborators  in  Philadelphia  (left)  and  Armonk,  New  York  (right)  by  viewing  three- 
dimensional  reconstructions  of  their  offices  (U  Penn). 


6.1.2  Omnidirectional  Geometry 


Being  aware  of  the  significance  of  representation  in  the  visual  algorithms  performance  we  espoused  the 
opinion  with  a  few  other  researchers  that  it  would  be  worth  realizing  such  representations  in  the  sensor 
level  rather  than  during  processing.  Our  work  (Daniilidis  and  co-workers)  in  the  image  formation  process 
was  strongly  motivated  bv  biological  findings.  Artificial  visual  systems  face  difficulties  in  tasks  like 
navigating  on  uneven  terrain  or  detecting  other  movements.  Paradoxically,  these  are  tasks  which  biological 
systems  like  insects,  with  very  simple  brains,  can  very  easily  accomplish.  The  difference  is  that  there  are 
hundreds  of  evolved  biological  eye  designs  but  only  one  artificial  camera:  the  CCD-chip  with  a  lens.  As  a 
prominent  and  useful  example,  we  chose  to  study  the  basic  questions  underlying  omnidirectional  sensing 
and  its  realization  with  catadioptric  designs  -  combinations  of  reflective  and  refractive  components.  Our 
motivation  for  a  unifying  representation  of  these  new  sensors  lead  (ICCV-99,  ECCV-2000)  to  the  discovery 
of  a  novel  geometric  theory  that  described  reflections  of  the  3D  world  on  mirrors  followed  by  projections 
on  the  omnidirectional  plane.  This  model  unifies  all  second  order  reflective  surfaces  and  includes  the 
traditional  CCD  camera  as  a  degenerate  case.  It  sheds  light  on  the  constraints  for  self-calibration  of  a 
camera  and  facilitates  new  projective  reconstructions  based  on  the  concepts  of  oriented  projective 
geometry.  Moreover,  in  collaboration  with  Hicks  (JMIV  2000)  we  established  robust  spatiotemporal 
signatures  for  mapping  and  localization  using  angular  measurements  picked  up  in  omnidirectional  images. 

Work  on  omnidirectional  vision  had  direct  impact  on  our  contribution  to  the  Tactical  Mobile  Robotics 
program.  During  the  October  2000  TMR-Demo  we  showed  how  the  Urbie  robot  can  climb  several  flights 
of  stairs  using  omnidirectional  vision  sensors.  Our  ongoing  work  includes  a  novel  sensor  design  being  able 
to  deliver  registered  and  fused  information  from  many  directions  and  in  three  bands  of  the  spectrum  (far  IR, 
near  IR,  visible). 


6.1.3  Camera  Motion  Estimation 


Based  on  the  analysis  techniques  described  above,  Tomasi  has  developed  new  video  processing  methods 
for  the  computation  of  three-dimensional  properties  of  the  scene  and  of  the  camera  motion.  He  proved 
previous  methods  to  provide  heavily  biased  estimates,  and  proposed  a  general  and  efficient  technique  for 
overcoming  bias  in  the  presence  of  outliers.  The  new  method  estimates  the  direction  of  camera  motion  from 
a  pair  of  consecutive  video  frames  with  an  accuracy  of  better  than  one  degree,  compared  to  about  fifteen- 
twenty  degrees  of  the  best  competing  methods  under  equal  circumstances.  These  advances  enable  the  use 
of  camera-driven  vehicles  and  three-dimensional  scene  reconstruction  in  realistic  scenarios,  for 
applications  in  automatic  surveillance,  site  inspection,  and  aerial  navigation  and  survey. 


Figure  9:  Computation  of  camera  motion.  In  spite  of  the  very  obvious  outliers  (circled,  left)  in  the 
image  motion  measurements,  the  direction  of  camera  motion  (right)  is  computed  with  less  than  one 
degree  of  error  (red  true,  blue  computed). 


6.1.4  Detection  of  Stereo  and  Motion  Discontinuities 

In  order  to  process  video,  it  is  important  to  segment  foreground  from  background,  and  more  generally 
different  objects  from  each  other.  The  motion  field  and  the  disparities  from  stereo  image  pairs  can  both 
provide  important  clues  for  this  segmentation.  We  (Tomasi)  have  developed  motion  and  disparity  based 
segmentation  methods  based  on  energy  minimization  techniques.  Our  algorithms  alternate  between 
segmenting  the  image  into  a  number  of  nonoverlapping  regions  and  finding  the  affine  parameters 
describing  the  displacement  function  of  each  region.  Experiments  on  real  images  show  the  algorithms 
ability  to  find  accurate  segmentations  and  displacement  maps,  as  well  as  discontinuities  and  creases,  from  a 
wide  variety  of  stereo  and  motion  imagery.  The  results  can  be  used  in  tracking  systems  that  focus  on  one  of 
many  people  in  a  scene,  or  in  splitting  foreground  from  background  for  algorithms  that  use  the  background 
to  determine  camera  motion  and  process  the  foreground  for  different  purposes. 


Figure  10:  Depth  and  motion  discontinuities.  Even  with  shallow  depths  (top  left),  depth 
discontinuities  (top  right)  are  crisp  and  accurate.  Features  moving  by  small  and  complex  motions 
(bottom  left)  are  accurately  segmented  (bottom  right)  into  foreground  and  background. 

6.1.5  The  Sensitivity  of  Structure  from  Motion 

In  spite  of  decades  of  research,  it  is  still  an  open  question  how  well  camera  motion  and  scene  geometry  can 
be  recovered  from  standard  video.  Tomasi  has  been  conducting  a  systematic  effort  to  answer  this  question 
in  a  quantitative  manner.  He  has  developed  a  mathematical  technique  that  allows  computing  the  probability 
density  of  parameter  estimation  errors  from  the  probability  density  of  image  noise.  He  has  also  extended 
Cramer-Rao  bounds  to  account  for  constraints,  making  this  powerful  predictive  tool  applicable  to  computer 
vision  problems.  These  techniques  have  made  it  possible  to  make  statements  like  the  following:  "video 
from  a  standard  television  camera  with  a  20mm  lens  in  front  of  a  scene  between  2  and  3  meters  away  can 
lead  to  camera  motion  estimates  that  are  accurate  within  1 .2  degrees  with  image  noise  of  one  pixel  standard 
deviation.”  Bias  and  probabilistic  skew  of  algorithms  can  be  calculated  as  well.  This  type  of  analysis  makes 
the  performance  of  vision  algorithms  predictable,  and  therefore  usable  in  a  wide  array  of  applications. 


6.1.6  Object  Recognition 

Taylor  has  developed  a  novel  approach  to  the  problem  of  object  recognition  which  exploits  a  new  method 
for  determining  whether  two  sets  of  image  points  could  be  projections  of  the  same  constellation  of  3D 
points  under  scaled  orthography.  This  method  shares  some  of  the  advantages  of  image  based  recognition 
schemes  in  that  all  computations  are  based  solely  on  image  measurements  and  no  information  about  the 
underlying  3D  structure  is  assumed.  However,  the  proposed  scheme  has  a  significant  advantage  over  such 
methods  in  that  it  is  able  to  correctly  recognize  images  of  objects  taken  from  viewpoints  other  than  those  in 
the  database.  Importantly,  the  method  avoids  the  combinatoric  issues  encountered  in  other  feature  based 
recognition  schemes  by  recasting  the  correspondence  problem  as  a  sorting  problem. 


Figure  2  Three-dimensional  mapping  for  supervised  autonomy.  A  robot  with  an  omnidirectional 
camera  explores  the  interior  of  an  abandoned  building  in  Fort  Sam  Houston  (left).  Omnidirectional 
images  (center)  are  used  to  automatically  reconstruct  the  interior  of  the  budding  (right).  (U  Penn). 


6.1.7  Accurate  measurement  of  material  reflectance  properties 

In  order  to  design  robust  vision  systems,  one  must  understand  the  Role  of  Light/illumination  and  its 
interaction  with  surfaces.  Towards  that  goal,  Bajcsy  and  Angelopoulou  have  designed  a  laboratory  with  a 
spectrograph  that  enables  us  to  measure  Bidirectional  Reflectance  Distribution  Function  (BRDF)  of 
surfaces  in  the  spectral  domain  with  resolution  of  0.7nm.  As  far  as  we  know,  this  is  the  highest  resolution 
available  currently  in  the  vision  community  and  we  plan  to  make  it  widely  available. 

This  facility  enables  us  to  measure  the  spectral  response  on  different  types  of  light  sources  and 
transmittance  of  various  color  filters.  We  developed  a  new  theory  of  invariance  of  incident  illumination, 
and  performed  initial  tests  to  verify  this  theory,  which  in  turn  will  help  us  to  separate  the  effect  of 
illumination  from  surface  reflectance  and  scene  geometry. 


6.1.8  Template  Matching  and  Neuroanatomy 

To  characterize  the  natural  anatomic  variability  of  the  human  brain  so  that  its  confounding  effect  can  be 
minimized  in  the  identification  of  brain  structures  as  revealed  on  high-resolution  medical  images,  a 
computational  framework  has  evolved  in  which  individual  anatomies  are  modeled  as  spatially  warped 
versions  of  a  canonical  representation  of  neuroanatomy,  known  as  a  brain  atlas  or  template.  The 
assumption  is  that  at  a  certain  spatial  scale  normal  instances  of  the  same  anatomy  share  a  common 
topological  scheme  and  that  differences  between  individuals  appear  only  as  variations  in  the  local  shape 
details. 

To  realize  this  framework,  Gee  and  Bajcsy  have  invented  the  method  of  elastic  matching  for  determining 
the  spatial  mapping  between  a  three-dimensional  (3-D)  image  pair  in  which  one  image  volume  is  modeled 
as  an  elastic  continuum  that  is  deformed  to  match  the  appearance  of  the  second  volume.  Remarkably,  in 
addition  to  deformable  representations  of  anatomy,  the  elastic  registration  technique  enables  many  useful 
applications.  An  MRI  (magnetic  resonance  imaging)  or  X-ray  CT  (computed  tomography)  study  can  be 
interpreted  using  knowledge — such  as  anatomic  labels  and  structural  boundaries — transferred  from  an  atlas 
once  the  mapping  is  computed  that  spatially  aligns  the  atlas  with  the  neuroanatomy  depicted  in  the  study. 
The  transformed  labels  automatically  localize  their  corresponding  structures  in  the  image  study. 


Visualization  is  achieved  by  deforming,  according  to  the  mapping,  the  structural  boundaries  contained  in 
the  atlas  and  then  displaying  the  result.  Quantitative  measurements  of  any  structure  in  the  study  can  be 
directly  inferred  from  the  morphometric  information  associated  with  its  warped  atlas  version.  If  functional 
scans  have  also  been  acquired,  the  physiologic  information  can  be  conveniently  related  through  the  same 
mapping  to  a  standard  anatomic  space  defined  by  the  atlas,  after  the  subject’s  pose  in  these  scans  is  made 
the  same  as  in  the  structural  images. 

A  major  thrust  of  current  research  has  been  to  construct  probabilistic  representations  that  share  the  practical 
benefits  of  the  deformable  modeling  paradigm  but  additionally  attempt  to  capture  the  statistics  governing 
biological  variability.  The  work  on  establishing  covariances  on  the  substructures  of  heterogeneous 
anatomy  entails  development  of  new  techniques  in  applied  statistics  and  has  important  consequences  for 
the  modeling  of  very  high-dimensional  structures. 


6.2  Motion  Synthesis  and  Planning 

6.2.1  Randomized  Motion  Planning 

The  complexity  of  motion  planning  is  known  to  grow  exponentially  in  the  number  of  degrees  of  freedom. 
Latombe  and  his  co-workers  have  developed  and  refined  successive  planning  algorithms  based  on  a 
random  sampling  scheme  of  the  configuration  space  of  the  moving  objects  (robots  or  any  kind  of  physical 
agents).  We  have  implemented  and  experimented  with  these  algorithms  on  many  examples  and 
applications.  Along  the  way  we  have  also  developed  important  techniques  needed  to  implement  random- 
sampling  planners  (e.g.,  collision  checking,  sampling  strategies,  space  indexing).  We  have  formally  proven 
that  under  generally  satisfied  assumptions  our  random-sampling  approach  finds  a  motion  path  with  high 
probability.  In  practice,  our  planners  have  shown  to  be  quite  fast. 

Our  work  on  randomized  planning  has  led  to  what  is  now  commonly  considered  as  the  only  available 
practical  approach  to  robot  path  planning  with  many  degrees  of  freedom.  Many  other  research  institutions, 
both  in  the  US  and  in  Europe,  have  students  and  researchers  working  on  improving  and/or  extending  our 
random-sampling  approach.  For  example,  at  MIT,  Prof.  Feron  uses  our  approach  to  generate  motion 
strategies  for  autonomous  helicopters;  at  Texas  A&M,  Prof.  Amato  uses  this  approach  to  predict  protein 
folding  motions.  Ourselves  have  applied  our  planning  methods  to  a  variety  of  problems,  including: 

■  Planning  of  radiosurgical  procedures  of  brain  tumors  for  a  new  system  (the  Cyberknife)  —  this  system 
is  now  in  clinical  use  at  Stanford  Medical  Center  and  is  marketed  by  Accuray,  Inc. 

■  Animation  of  autonomous  digital  actors  —  a  company,  The  Motion  Factory,  was  formed  to  market 
motion  planning  software  for  digital  actors. 

■  Prediction  of  ligand-protein  binding  motions  in  order  to  select  drug  candidates  from  molecule 
databases  -  this  work  was  initially  funded  by  a  grant  from  Pfizer  and  is  now  continued  in  cooperation 
with  the  Biochemistry  Department  as  part  of  the  Stanford's  Bio-X  initiative. 

We  recently  obtained  a  5-year  grant  from  General  Motors  to  adapt  our  motion  planning  methods  to 
manufacturing  problems. 


Figure  3  The  Cyberknife  System  for  Computer  Assisted  Surgery  (left)  and  the  use  of  randomized 
motion  planning  algorithms  for  surgical  planning  (middle,  right).  (Stanford). 


Figure  4  Biomolecular  modeling,  planning  and  simulation  algorithms  from  projects  leveraged  from 
MURI  funding  (Stanford). 


6.2.2  Motion  Planing  with  Visibility  Constraints 

In  visibility-based  planning,  Latombe  and  Guibas  have  developed  entirely  new  motion  planning  methods  to 
allow  an  agent  to  move  in  a  physical  space  (without  collision)  in  order  to  achieve  sensing  goals  (for 
instance,  map  a  new  environment,  find  a  hiding  target,  track  an  evading  target).  Our  methods  typically 
search  an  information  space  built  on  top  of  the  agent’s  configuration  space.  They  combine  two  types  of 
computational  geometry  techniques:  collision  checking  and  visibility  analysis.  Two  major  methods 
developed  are  a  next-best-view  planner  to  efficiently  build  the  map  of  a  new  environment  and  an 
environment-sweep  planner  to  find  and  locate  moving  targets. 

Probabilistic  road-map  (PRM)  planners  have  shown  great  promise  in  attacking  motion  planning  problems 
with  many  degrees  of  freedom  that  were  previously  infeasible.  Yet  when  such  a  planner  fails  to  find  a  path, 
it  is  not  clear  that  no  path  exists,  or  that  the  planner  simply  did  not  sample  adequately  or  intelligently  the 
free  part  of  the  configuration  space.  We  propose  to  attack  the  motion  planning  problem  from  the  other  end, 
focussing  on  disconnection  proofs,  or  proofs  showing  that  there  exists  no  solution  to  the  posed  motion 
planning  problem.  Just  as  PRM  planners  avoid  generating  a  complete  description  of  the  configuration 
space,  our  disconnection  provers  search  for  certain  special  classes  of  proofs  that  are  compact  and  easy  to 
find  when  the  motion  planning  problem  is  ‘obviously  impossible,’  avoiding  complex  geometric  and 
combinatorial  calculations.  We  have  demonstrated  such  a  prover  in  action  for  a  simple,  yet  still  realistic, 
motion  planning  problem.  Failure  of  the  prover  suggests  key  milestones,  or  configurations  of  the  robot  that 
can  then  be  passed  on  and  used  by  a  PRM  planner.  Thus  by  hitting  the  motion  planning  problem  from  both 
ends,  we  can  resolve  the  existence  of  a  path,  except  in  truly  delicate  border-line  situations. 


6.2.3  Sensor  Based  Motion  Planning 


Kumar  and  Mintz  have  developed  a  framework  for  motion  planning  that  incorporates  set-valued  estimates 
of  uncertainty  at  different  levels  of  granularity.  These  algorithms  yield  the  best  case  performance  under  the 
worst  possible  uncertainty,  without  any  assumptions  on  the  underlying  distribution. 


We  have  developed  algorithms  for: 

■  open  loop  motion  planning  -  the  robot(s)  plan  the  optimal  path  from  a  start  position  to  a  goal  position 
based  on  the  information  available  at  the  start  position; 

■  recursive  open  loop  motion  planning  -  the  robot(s)  plan  the  optimal  path  from  a  start  position  to  a  goal 
position  based  on  the  information  available  at  the  start  position,  but  this  path  is  refined  as  additional 
information  becomes  available; 

■  closed  loop  motion  planning  -  the  robot(s)  chose  the  optimal  feedback  policy  (a  control  law  that  relates 
real  time  sensory  information  to  actuator  inputs)  from  a  start  position  to  a  goal  position  based  on  the 
information  available  at  the  start  position;  and 

■  recursive  closed  loop  motion  planning  -  the  robot(s)  chose  the  optimal  feedback  policy  (a  control  law 
that  relates  real  time  sensory  information  to  actuator  inputs)  from  a  start  position  to  a  goal  position  and 
this  path  is  refined  as  additional  information  becomes  available. 

Our  main  contributions  have  been  threefold.  First,  we  have  established  saddle  point  properties  for  open 
loop  motion  plans  in  a  wide  class  of  problems  thus  guaranteeing  global  optimality.  Second,  we  have 
developed  algorithms  for  developing  piecewise  linear  closed  loop  motion  plans.  We  have  shown  that  at 
least  two  linear  feedback  laws  are  necessary  to  circumvent  a  single  obstacle  and  have  established  an  upper 
bound  on  the  number  of  switches  required  in  the  general  case.  Finally,  we  have  algorithms  that  generate 
min  max  solutions  for  ail  four  classes  of  strategies. 

6.2,4  Sensor  Fusion  for  Tracking  and  Interception  of  Moving  Targets 

Mintz  and  his  students  have  solved  a  set  of  important  problems  that  are  fundamental  to  sensor  fusion  for 
control  and  planning  of  autonomous  robots. 

■  We  have  solved  the  set-valued  (optimal  fixed-geometry  confidence  set)  problem  for  an  extremely  wide 
range  of  sampling  distributions.  The  class  of  distributions  includes:  unimodal,  multimodal  probability 
density  functions,  which  may  be  symmetric  or  non-symmetric,  and  possessing  heavy-tailed  behavior. 
The  optimal  solutions  to  these  decision  problems  are  either  continuous  monotone  piecewise  linear 
rules  with  alternating  sections  of  unit  and  zero  slope,  or  non-monotone  rules  which  can  be  very 
accurately  approximated  as  piecewise  linear  rules  with  sections  where  the  slope  is  greater  than  unity  or 
is  zero. 

■  The  multivariate  versions  of  the  confidence  set  problem  have  been  solved  when  the  component  noise 
distributions  are  independent.  If  the  noise  components  exhibit  dependencies,  highly  accurate 
approximate  solutions  are  described  which  are  appropriate  in  all  but  the  most  extreme  cases.  For 
example,  in  the  case  of  a  multivariate  normal  distribution,  if  the  correlation  coefficients  are  less  than 
0.9  in  absolute  value,  then  we  obtain  results  that  are  within  3  percent  of  optimal. 

■  For  the  dynamic  version  of  the  confidence  set  problem,  we  have  obtained  optimal  results  which  impose 
virtually  no  limits  on  the  dynamic  models,  and  only  impose  an  independence  assumption  on  the 
additive  sensor  noise. 

■  We  have  applied  the  theory  to  a  practical  industrial  application  using  multiresolution  sensors  and 
verified  that  extremely  good  correspondence  between  the  theoretical  (predicted)  performance  and  the 
empirically  obtained  (actual)  performance  was  achieved. 


All  of  these  results  make  use  of  game-theoretic  solutions  and  robust  statistical  inference  techniques  which 
we  have  developed  over  the  past  several  years. 
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Figure  5  Sensor  fusion  for  localization  and  control  while  climbing  stairs.  The  ISR  robot  climbing 
stairs  (left).  Different  estimates  of  the  heading  angle  provided  by  the  acoustic  and  vision  sensors 
(center)  and  their  confidence  measures  (right).  (U  Penn). 


6.3  Motion  Execution  and  Control 

6.3.1  Control  and  Coordination  of  Multiple  Robots 

Kumar  and  Ostrowski  have  developed  a  framework  for  the  control  of  a  team  of  robots  navigating  in  a 
terrain  with  obstacles,  while  maintaining  a  desired  formation  and  changing  formations  when  required.  This 
problem  is  formulated  by  modeling  the  team  as  a  triple  consisting  of  a  group  element  that  describes  the 
gross  motion  of  the  team,  a  set  of  shape  variables  that  describe  the  relative  positions  of  robots,  and  a  control 
graph  that  describes  the  behaviors  of  the  robots  in  the  formation.  This  model  captures  the  different  levels  of 
coordination  required  for  such  tasks.  For  example,  at  the  lowest  level,  it  is  necessary  for  each  robot  to 
control  its  motion,  to  avoid  collisions  with  its  neighbors,  and  to  move  along  a  desired  trajectory.  This  is 
captured  by  the  shape  variables  for  the  formation.  At  an  immediately  supervisory  level  it  is  necessary  to 
outline  a  strategy  for  maintaining  a  formation.  In  a  system  of  two  robots,  this  might  be  as  simple  as  a 
leader-follower  strategy  where  a  leader  plans  and  follows  a  preferred  trajectory  while  the  follower 
maintains  a  specified  relative  position  and  orientation  with  respect  to  the  leader.  The  control  laws  may  be 
specially  designed  for  manipulation  where  the  two  robots  grasp  and  carry  a  payload  cooperatively.  When 
the  number  of  robots  is  more  than  two,  there  might  be  more  than  one  leader-follower  pairs  or  a  more 
complex  structure  of  interaction.  This  discrete  level  of  coordination  is  captured  by  the  control  graph.  In 
contrast  to  the  control  laws  for  individual  robots  where  the  state  variables  and  inputs  change  continuously, 
there  are  only  a  finite  number  of  distinct  formations  and  the  changes  in  formations  represent  discrete 
changes  in  the  organization  or  the  control  graph  for  the  team.  Finally,  at  the  highest  level,  it  is  necessary  to 
plan  the  trajectory  for  the  entire  team  of  robots  based  on  the  available  terrain  information.  This  essentially 
boils  down  to  finding  the  optimal  trajectory,  g(t)  €  SE( 2). 

We  have  used  mathematical  tools  from  graph  theory,  differential  geometry  and  control  theory  to  solve  what 
is  essentially  a  problem  in  hybrid  systems.  We  have  characterized  the  different  possible  formations  for  a 
team  of  n  robots  by  enumerating  the  possible  control  graphs.  This  includes  a  proof  for  the  upper  bound  on 
the  total  number  of  possible  control  graphs  and  an  enumeration  of  the  equivalence  classes  of  mixed  graphs 
that  are  relevant  to  this  problem.  The  gross  motion  is  obtained  using  tools  from  Lie  theory  that  allow  us  to 
exploit  the  Lie  structure  of  the  Euclidean  group.  Finally,  we  developed  a  discrete  coordination  strategy  for 
changing  formations  in  terms  of  a  sequence  of  elementary  transitions  in  the  control  graph  that  satisfy  the 
constraints  in  the  problem. 


Figure  6  Vision  guided  cooperative  control  of  robots.  Two  robots  maintain  formation  while  avoiding 
obstacles  (left)  and  a  representative  omnidirectional  image  from  the  lead  robot  in  which  it  can  see  the 
follower  (right).  (U  Penn). 

The  framework  and  the  algorithms  are  relevant  to  any  scenario  in  which  it  is  necessary  to  coordinate  a  large 
number  of  networked  embedded  systems.  For  example,  in  military  applications,  vehicles  are  often  required 
to  maintain  a  close  formation,  and  in  autonomous  vehicle  highway  systems,  there  is  a  need  to  control 
platoons  of  cars  in  formation  while  allowing  the  cars  to  perform  such  maneuvers  as  lane  changes,  merges, 
and  avoiding  obstacles  on  the  road.  In  the  DARPA  Tactical  Mobile  Robotics  program,  considerable 
emphasis  is  being  placed  on  the  coordination  of  autonomous  robots  in  search  and  rescue  missions, 
collaborative  scouting  and  reconnaissance,  and  in  three-dimensional  mapping.  Our  ideas  have  direct 
application  in  these  settings. 


6.3.2  Visual  Servoing 

Taylor  and  Ostrowski  have  worked  on  the  problem  of  controlling  the  spatial  position  and  orientation  of  a 
robotic  platform  based  on  the  image  data  obtained  from  a  video  camera  mounted  on  that  platform.  More 
specifically,  we  proposed  control  laws  that  generate  translational  and  angular  velocities  that  would  cause 
the  robot  to  achieve  and  maintain  a  fixed  position  and  orientation  with  respect  to  a  set  of  feature  points  in 
the  scene. 

The  proposed  control  schemes  make  use  of  well  established  techniques  for  computing  estimates  for  the 
relative  orientation  of  two  camera  positions  from  a  set  of  feature  correspondences.  An  important  advantage 
of  these  control  schemes  is  that  it  is  possible  to  demonstrate  analytically  that  they  are  globally  convergent 
even  in  the  presence  of  large  calibration  errors  in  both  the  intrinsic  parameters  of  the  camera  and  in  the 
extrinsic  parameters  which  relate  the  frame  of  reference  of  the  camera  to  the  body  frame  of  the  robot 
platform  which  is  being  controlled  Furthermore  no  a’priori  knowledge  about  the  structure  of  the  scene  is 
assumed. 

The  research  carried  out  under  the  aegis  of  MURI  funding  has  opened  up  a  number  of  further  research 
opportunities  and  collaborations.  Taylor’s  research  on  real-time  vision  based  guidance  systems  carried  out 
under  this  grant  has  been  incorporated  into  a  project  funded  by  Delphi  Automotive  to  develop  real  time 
vision  based  lane  tracking  systems  for  advanced  collision  warning  systems.  The  research  on  Vision  Based 
human  computer  interfaces  has  led  to  new  ideas  on  how  to  develop  flexible  interface  systems  that  can  be 
customized  for  disabled  individuals.  This  work  will  be  pursued  under  a  recently  awarded  NSF  grant.  Our 
efforts  on  vision  based  control  of  teams  of  mobile  robots  has  led  to  our  participation  in  the  DARPA  funded 
MARS  program  and  has  sparked  a  new  effort  on  collaborative  robotics  in  the  laboratory. 

6.3.3  Robust  Control  of  Remote  Systems  in  the  Presence  of  Communication  Delays 

Bajcsy  and  her  coworkers  have  developed  a  robust  motion  control  algorithm  for  the  appropriate  treatment 
of  long  latency  times.  Delays  are  the  most  crucial  problem  in  controlling  robot  vehicles  over  the  network. 


Our  algorithm  is  based  on  (i-synthesis  based  control  and  outperforms  the  known  algorithms  of  the 
literature.  This  work  was  conducted  in  close  collaboration  with  Latombe  and  his  students  at  Stanford. 


6.3.4  Control  of  Advanced  Locomotion  Systems 

Ostrowski  and  his  cow'orkers  study  of  motion  planning  and  control  for  dynamic  mobile  robots.  Examples  ^ 
from  this  class  that  have  been  studied  include  a  blimp  robot,  an  eel-like  robot,  and  multiple  walking  (Sony  s 
legged  robot)  and  wheeled  robots.  The  blimp  robot  provides  an  excellent  example  of  an  underactuated 
mechanical  system,  and  has  involved  the  study  of  very  fundamental  issues  of  trajectory  tracking  (e.g.,  how 
do  you  parallel  park  a  blimp?).  There  have  also  been  important  contributions  made  to  the  area  of  dynamic 
visual  servoing,  where  the  dynamics  of  the  robot  are  embedded  in  the  sensor  space  (here  the  vision  plane). 
This  allows  one  to  properly  control  a  dynamic  system,  such  as  planes,  blimps,  helicopters,  etc.,  using  visual 
feedback.  The  eel  robot  has  served  to  study  biomimetic  principles  applied  to  the  design  of  amphibious 
(land-based  and  underwater)  robots.  We  have  developed  a  methodology  for  developing  control  laws  for 
such  dynamic  robots,  and  a  hierarchical  architecture  that  is  used  to  solve  the  full  motion  planning  problem 
of  building  controllers  to  move  to  any  desired  location.  The  strategy  has  been  to  rely  on  a  decomposition 
of  the  system  into  multiple  layers  of  abstraction,  each  of  which  simplifies  the  control  and  motion  planning 
problem.  Finally,  we  have  been  performing  work  on  the  coordination  and  control  of  multiple  autonomous 
robots.  This  has  been  preformed  in  parallel  on  two  separate  platforms,  with  the  goal  of  understanding  the 
trade-offs  and  benefits  of  developing  modular  code  that  can  be  applied  to  both  platforms.  The  wheeled 
robots  have  been  used  to  study  formation  control  as  well  as  issues  in  multi-robot  grasping.  The  legged 
robots  have  primarily  been  used  to  study  formation  control  and  decision-making  strategies  for  cooperative 
robots  competing  in  robotic  soccer  through  our  involvement  in  the  international  RoboCup  competitions. 


Our  work  has  been  at  the  forefront  of  several  areas  of  modem  research.  In  visual  servoing,  we  have 
proposed  two  new  paradigms-  that  of  dynamic  visual  servoing,  where  the  dynamics  of  the  system  are 
directly  incorporated  into  the  vision-based  control;  and  visual  motion  planning,  where  motion  plans  are 
developed  in  the  sensor  space  (visual  plane)  rather  than  with  respect  to  some  inertial  frame.  These  tools 
have  important  implications  for  use  in  autonomous  robot  exploration  and  tracking,  using  dynamic  mobile 
robots  such  as  planes,  blimps,  spacecraft,  or  helicopters.  Through  the  support  of  this  MURI,  our  work  has 
led  to  two  new  projects  targeted  towards  extended  duration  aerial  vehicles.  The  first  is  a  project  with 
Carnegie  Mellon  University  towards  developing  "infinite"  (very  long)  duration  blimps,  used  for 
surveillance,  environmental  monitoring,  etc.  The  second  is  a  project  we  are  starting  up  with  Global 
Aerospace  to  develop  a  control  methodology  for  a  global  network  of  balloons.  At  Penn,  there  is  also  a 
newly  established  project  to  build  a  30  ft.  outdoor  blimp  to  be  flown  autonomously.  Lastly,  we  are  in  the 
planning  stages  right  now  to  kick  off  an  autonomous  aerial  vehicle  project  that  seeks  to  develop  a  sensor 
fusion  approach  (using  vision  as  a  central  sensor)  towards  formation  flight  of  airplanes. 


Figure  7  Examples  of  novel  locomotion  systems:  A  modular,  robotic  eel  that  is  able  to  swim  under 
water  (left);  and  an  indoor,  vision-guided  blimp  (right),  (U  Penn). 

Our  work  on  the  eel  has  provided  new  insights  into  underwater  and  amphibious  locomotion,  with 
applications  to  covert  operations  and  surf-zone  counter-mine  and  de-mining  efforts.  The  hierarchy 
developed  while  working  on  the  eel  is  applicable  to  a  wide  range  of  dynamic  locomotion  systems.  The 
general  methodology  of  working  with  a  modular  robot  has  also  led  to  an  NSF  CAREER  award  targeted 
towards  the  study  of  modular  locomotion  systems.  The  final  area  of  work  on  multiple  mobile  robots  has 
led  to  funding  through  the  DARPA  MARS  initiative,  and  sponsorship  by  Sony  Corporation  to  compete  in 
the  Sony  legged  league  of  the  RoboCup  competition. 


6.3.5  Collision  Detection  using  Kinetic  Data  Structures 


Computer  systems  commonly  cache  the  values  of  variables  to  gain  efficiency.  In  applications  where  the 
goal  is  to  track  attributes  of  a  continuously  moving  or  deforming  physical  system  over  time,  caching 
relations  between  variables  works  better  than  caching  individual  values.  The  reason  is  that,  as  the  system 
evolves,  such  relationships  are  more  stable  than  the  values  of  individual  variables.  Kinetic  data  structures 
(KDSs)  are  a  novel  formal  framework  for  designing  and  analyzing  sets  of  assertions  to  cache  about  the 
environment,  so  that  these  assertion  sets  are  at  once  relatively  stable  and  tailored  to  facilitate  or  trivialize 
the  computation  of  the  attribute  of  interest.  Formally,  a  KDS  is  a  mathematical  proof  animated  through 
time,  proving  the  validity  of  a  certain  computation  for  the  attribute  of  interest.  KDSs  have  rigorous 
associated  measures  of  performance  and  their  design  shares  many  qualities  with  that  ot 
classical  data  structures. 

The  KDS  framework  has  led  to  many  new  and  promising  algorithms  in  graphics  and  robotics  applications. 
Specific  accomplishments  include  new  algorithms  for  tracking: 

■  Extent  Attributes:  convex  hull,  width,  diameter,  etc.  among  moving  points 

■  Proximity  Attributes:  closest  pair,  Voronoi/Delaunay  diagrams,  etc.  among  moving  points,  polygons, 
or  balls;  local  environment  of  moving  ‘agents’ 

■  Space  Partitions:  BSPs  for  occlusion  culling  among  moving  triangles 

■  Optimization  Problems:  MSTs  for  moving  points  or  graphs  with  variable  edge  weights;  linear 
programming 

■  Visibility:  visibility  with  moving  observers  and  obstacles 
*  Collision  Detection:  for  rigid  and  deformable  objects 

■  Routes  and  Locations:  of  mobile  stations  in  ad-hoc  networks 

A  key  feature  of  all  these  algorithms  is  that  they  sample  parts  of  the  system  under  motion  only  as  needed  to 
maintain  the  desired  attribute(s).  These  leads  to  algorithms  that  are  efficient,  robust,  and  often  easily 
scalable,  as  the  computations  involved  are  local.  KDSs  are  a  completely  new  research  area  that  came  to  be 
primarily  because  of  this  MURI  funding. 


Figure  8  —  A  and  B  represent  ongoing  work  on  protein  folding  with  the  goal  of  simulating  the  effect  of  the 
solvent  molecules  (water)  by  a  set  of  potentials  and  avoiding  their  explicit  modeling  in  the  simulation 
(Stanford). 

7  Interactions  with  Private  Industry 

As  a  group,  the  MURI  researchers  have  close  collaborations  and  sponsorship  from  several  industrial 
organizations.  A  list  of  industrial  partners  that  are  involved  in  MURI  related  activities  is  provided  below. 


1 .  Advanced  Networks  and  Services,  Inc. 

2.  Applied  Resource  Corporation 

3.  Cytometrics,  Inc. 

4.  Deneb  Robotics 

5.  General  Motors,  GM  Research  Labs 

6.  General  Reality  Company 

7.  Global  Aerospace 

8.  Honda  Motors,  Tochigi  Research  Center 

9.  Intel  Corporation,  Microprocessor  Research  Laboratory 

10.  Interval 

11.  Microsoft  Corporation 

12.  Pfizer  Pharmaceuticals 

13.  Siemens  Research  Corporation  in  Princeton,  NJ 

14.  Sony  Corporation 

15.  ST  Microelectronics 

16.  Tokheim  Corporation 


Figure  9  Results  from  leveraged  projects.  A  wheelchair  augmented  with  robotic  legs  for  superior 
traction  and  for  overcoming  obstacles  (left);  An  inexpensive  feeder  being  used  by  an  individual  with 
a  C-6  lesion  (center);  and  (c)  A  robot  arm  for  children  with  cerebral  palsy  with  a  “game  boy”  like 
interface  (U  Penn). 

We  provide  below  several  examples  of  our  collaborations  with  industry. 

■  Initial  demonstrations  developed  under  the  MURI  research  has  enabled  Tomasi  to  request  and  obtain 
funding  from  Intel  in  Santa  Clara,  CA,  for  the  development  of  a  PC-based  system  for  American  Sign 
Language  recognition.  This  funding  comes  under  a  three-year  agreement,  and  involves  both  student 
support  and  equipment. 

■  Additionally,  initial  demonstrations  developed  under  the  MURI  research  has  enabled  Guibas,  Latombe, 
Motwani,  and  Tomasi  to  request  and  obtain  an  equipment  gift  from  Intel  in  Santa  Clara,  CA,  for  a 
project  entitled  "Motion  Representations  and  Algorithms".  This  gift  consists  of  5  Pentium  II-based  PC 
equipped  with  graphics  cards. 

■  Researchers  at  the  Intel  Microprocessor  Research  Labs  are  using  Kinetic  Data  Structure  ideas  to 
speed  up  interactive  computer  animations,  such  as  games.  An  Intel  researcher  is  implementing 
collision  detection  algorithms  based  on  work  recently  completed  by  Guibas  and  his  students.  Intel  is 
supporting  this  research  of  Guibas  by  providing  the  stipend  for  a  student,  some  summer  salary  support, 
and  equipment.  A  renewal  of  this  grant  is  pending. 

■  Research  developed  under  MURI  has  attracted  a  joint  three-year  grant  funded  by  Sony,  Intel,  and 
Interval.  This  multi-million  dollar  grant  is  managed  by  Prof.  Pat  Hanrahan  (Stanford),  and  aims  at 
developing  immersive  television  applications.  Tomasi  is  responsible  for  motion  and  shape  acquisition 


through  computer  vision  techniques.  The  grant  funds  research  by  five  faculty  members  and  ten 
graduate  students  at  Stanford. 

The  research  conducted  by  Latombe  and  Motwani  on  randomized  path  planning  has  enabled  us  to 
establish  a  collaboration  with  General  Motors  and  Deneb  Robotics.  General  Motors  supports  our 
research  on  finding  the  quasi-optimal  location  of  a  robot  through  SIMA  (the  Stanford  Industrial 
Manufacturing  Association).  Deneb  Robotics  has  given  us  a  free  license  of  their  robot  simulation 
software  IGRIP. 

A  cooperation  is  in  progress  between  Tomasi  and  the  General  Reality  Company  (GRC)  in  San  Jose, 
CA,  a  developer  of  hardware  and  software  for  3D  computer  graphics  and  3D  interactive  devices. 
Research  started  under  MURI  is  being  further  developed  at  GRC  to  measure  camera  motion  and  to 
reconstruct  the  shape  of  3D  scenes  and  objects  for  virtual  reality  applications. 

Ostrowski’s  work  on  control  of  nonholonomic  locomotion  systems  has  led  to  a  project  with  Global 
Aerospace  to  develop  a  control  methodology  for  a  global  network  of  balloons. 

Researchers  at  Microsoft  Research  are  interested  in  using  Guibas’  Kinetic  Data  Structures  for 
maintaining  visibility  among  moving  objects. 

The  research  conducted  by  Latombe  and  Motwani  on  randomized  path  planning  was  adapted  to 
develop  software  aimed  at  checking  whether  given  ligands  (small  flexible  molecules)  are  likely  to  bind 
against  given  protein  cavities.  This  work  was  done  under  a  broader  research  contract  with  Pfizer 
Pharmaceuticals  covering  other  aspects  of  computational  drug  design. 

Taylor  has  worked  with  Honda  Motors  at  Honda's  Tochigi  Research  Center  in  Japan  as  part  of  the 
Berkeley  activity  on  Vision-Based  Control  of  Autonomous  Vehicles.  Honda  Motors  have  active 
collaborations  with  Ostrowski,  Kumar  and  Taylor. 

General  Motors  is  funding  Bajcsy,  Kumar,  Taylor,  and  Ostrowski’ s  work  on  problems  related  to 
sensing  and  control  for  automated  vehicle  highway  systems. 

Tokheim  Corporation  sponsored  research  on  vision  based  control  of  robots  for  automated  fueling  of 
passenger  automobiles.  We  jointly  developed  an  experimental  prototype  consisting  of  a  robot,  a  set  of 
cameras,  and  a  suite  of  algorithms  for  control  and  sensing. 

Kumar  has  a  close  collaboration  with  Applied  Resource  Corporation  on  the  development  of  a  virtual 
prototyping  environment  to  design  robotic  aids  for  people  with  disabilities.  He  is  also  working  on  a 
new  collaboration  on  a  semi-automated,  ultra-mobile  wheelchair  system  for  people  confined  to 
wheelchairs. 

Our  work  on  path  optimization  and  finding  the  location  of  a  robot's  base  has  been  done  in  interaction 
with  GM  Research  Labs  in  Warren  (MI).  We  gave  one  presentation  at  GM  Research  Labs. 

The  application  of  PRM  to  Ligand  Binding  was  presented  to  several  visitors  from  pharmaceutical 
companies  including  Merck  and  Rhones-Poulenc. 

Bajcsy  and  Daniilidis  are  involved  in  the  National  Tele-immersion  Initiative  sponsored  by  Advanced 
Network  and  Services,  Inc.,  NYC.  The  GRASP  lab  demonstrated  the  world- wide  first  3D- 
Teleconferencing  and  is  actively  collaborating  with  the  above  mentioned  company,  Brown  University, 
and  the  University  of  North  Carolina. 

Daniilidis  applied  principles  studied  in  the  Algorithmics  of  Motion  in  a  collaboration  project  with 
Siemens  Research  Corporation,  Princeton,  NJ.  In  this  project,  industrial  plants  are  reconstructed 
from  multiple  views. 


■  The  Microprocessor  Research  Laboratory  (MRL)  of  Intel  is  interested  in  Kinetic  Data  Structures 
work  and  researchers  from  MRL  are  currently  involved  in  implementation  efforts  mentioned  above. 

■  We  have  recently  attracted  the  donation  of  several  experimental  robots  from  Sony  Corporation,  and 
will  be  working  on  algorithms  for  vision-based  motion  planning  of  small,  legged  robots. 

■  General  Motors  is  funding  Latombe  to  adapt  randomized  motion  planning  to  industrial  manufacturing 
problems,  in  particular  the  programming  of  spot-welding  robots. 

■  Work  developed  under  this  MURI  grant  has  led  to  a  cooperation  between  Tomasi  and  ST 
Microelectronics  for  the  automatic  motion  segmentation  in  video.  The  aim  of  this  grant  is  to  develop  a 
chip  to  be  used  in  television  set-top  boxes  for  various  video  enhancement  purposes. 

■  Tomasi  has  established  a  collaboration  with  Honda  for  the  development  of  image  motion  analysis 
algorithms  to  be  used  for  Honda’s  P3  humanoid  robot. 


8  Honors,  Awards,  and  Accomplishments 

8.1  Our  Students 

One  of  the  very  important  tasks  we  perform  is  training,  educating,  and  mentoring  graduate  students  who 
then  go  on  to  important  positions  in  academic,  government,  and  industrial  institutions.  We  listed  below 
examples  of  such  students  and  their  achievements. 


■  Stan  Birchfield  (Stanford  Ph.  D.),  has  joined  Quindi,  a  startup  company  in  Palo  Alto  developing  image 
processing  techniques  based  on  motion  analysis. 

•  Luca  Bogoni  (Penn  Ph.D.)  is  working  at  Samoff  SRI  research  Corp.  on  DARPA  sponsored  contracts. 

•  Jaydev  Desai  (Penn,  Ph.D.)  worked  with  Professor  Howe  (Harvard  University)  as  a  post-doctoral 
fellow  on  a  MURI  on  bio-mimetic  robots  and  is  currently  an  Assistant  Professor  at  Drexel  University. 

•  Hany  Farid  (Penn,  Ph.D.)  is  currently  an  Assistant  Professor  (NSF  Career  Award  winner)  at  Dartmouth 
University. 

■  David  Hsu  (Stanford,  Ph.D.)  has  joined  Compaq  Research  Lab  in  Cambridge,  MA. 

•  Lydia  Kavaraki  (Stanford  Ph.D.)  is  an  Assistant  Professor  (NSF  Career  Award  winner)  at  Rice 
University.  She  was  recently  awarded  a  Sloan  Fellowship. 

■  Jana  Kosceka  (Penn  Ph.D.)  worked  with  S.  Sastry  on  researched  sponsored  by  DoD,  including  a  MURI 
grant.  She  is  currently  an  Assistant  Professor  at  George  Mason  University. 

•  James  Kuffner  (Stanford,  Ph.D.)  is  working  at  the  University  of  Tokyo  under  Prof.  Inoue  to  apply 
random  planning  techniques  to  humanoid  robots. 

■  Venkat  Krovi  (Penn,  Ph.D.)  accepted  an  offer  from  McGill  University,  one  of  the  top  two  or  three 
universities  in  Canada.  He  won  a  prestigious  student  paper  competition  in  1997,  sponsored  by  AMR. 

•  Steve  Lavalle  (Post-doc,  Stanford)  is  currently  a  faculty  member  (NSF  Career  Award  winner)  at  Iowa 
State  University. 


■  Robert  Mandelbaum  (Penn,  Ph.D.)  is  working  at  Samoff  SRI  Research  Corp.  on  DARPA  sponsored 
contracts. 

■  Peng  Song’s  (Penn)  paper  with  Kumar  was  one  of  three  finalists  for  the  best  paper  award  at  the 
International  Conference  in  Robotics  and  Automation. 

■  Tom  Sugar  (Penn.  Ph.D.)  is  now  an  Assistant  Professor  at  Arizona  State  University. 

■  Milos  Zeffan  (Penn,  Ph.D.)  is  now  an  Assistant  Professor  at  the  University  of  Chicago. 

■  Hong  Zhang  (Penn,  Ph.D.)  is  currently  an  Assistant  Professor  at  Rowan  College  in  Rowan,  NJ. 

■  John  Zhang  (Stanford  Ph.  D.)  is  currently  working  at  SUN  Microsystems. 

■  Tong  Zhang  (Stanford  Ph.D.)  is  currently  a  researcher  at  the  IBM  T.  J.  Watson  Research  Center. 


8.2  Our  Faculty 

•  Leonidas  J.  Guibas  was  elected  as  ACM  Fellow. 

■  James  Ostrowski  has  been  appointed  as  an  Associate  Editor  of  the  IEEE  Control  Systems  Society 
Conference  Editorial  Board.  He  received  a  1999  NSF  CAREER  award  for  his  work  on  "Hybrid 
Locomotion  Systems  for  Varying  Terrains  and  Environments",  which  has  direct  relationship  to  the 
MURI's  aim  of  "Algorithmics  of  Motion",  in  developing  algorithms  for  controlling  new  types  of 
locomotion  systems  operating  in  complex  ("non- laboratory")  environments. 

•  Camillo  Taylor  received  a  NSF  CAREER  Award  in  1999  for  his  work  on  three-dimensional 
reconstruction  from  two-dimensional  Images  and  vision-based  control. 

■  James  Ostrowski  has  been  re-appointed  as  an  assistant  professor.  This  process  of  reappointment  is  the 
first  review  after  a  three  year  period.  The  second  three  year  review  results  in  tenure. 

•  Ruzena  Bajcsy  was  named  the  Assistant  Director  of  the  Computer  and  Information  Science  and 
Engineering  (CISE)  directorate  at  the  National  Science  Foundation  after  a  nationwide  search. 

■  Ruzena  Bajcsy  was  elected  to  the  National  Academy  of  Engineering  in  1998. 

■  Kostas  Daniilidis  chaired  the  IEEE  Workshop  on  Omnidirectional  Vision,  Hilton  Head  island.  June  12, 

2000. 
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